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ABSTRACT 

We  introduce  a  new  exemplar-based  inpainting  algorithm  that 
represents  the  region  to  be  inpainted  as  a  sparse  linear  combi¬ 
nation  of  example  blocks,  extracted  from  the  image  being  in- 
painted  or  an  external  training  image  set.  This  method  is  con¬ 
ceptually  simple,  being  computed  by  minimization  of  a  sim¬ 
ple  functional,  and  avoids  the  complexity  of  correctly  order¬ 
ing  the  filling  in  of  missing  regions  of  other  exemplar-based 
methods.  Initial  performance  comparisons  on  small  inpaint¬ 
ing  regions  indicate  that  this  method  provides  similar  or  better 
performance  than  other  recent  methods. 
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1.  INTRODUCTION 


or  from  a  separate  training  image  set.  These  image  blocks 
are  overlapped  to  reduce  blocking  artifacts,  and,  more  impor¬ 
tantly,  to  enable  information  from  the  exterior  of  the  inpaint¬ 
ing  region  to  propagate  to  blocks  entirely  within  the  interior, 
as  illustrated  in  Fig.  1. 


Exemplar  based  methods  are  becoming  increasingly  popular 
for  problems  such  as  denoising  [1,  2],  superresolution  [3,  4, 
5],  texture  synthesis  [6],  and  inpainting  [7,  8].  The  com¬ 
mon  theme  of  these  methods  is  the  use  of  a  set  of  actual  im¬ 
age  blocks,  extracted  either  from  the  image  being  restored, 
or  from  a  separate  training  set  of  representative  images,  as 
an  image  model.  In  the  case  of  inpainting,  the  approach  is 
usually  to  progressively  replace  missing  regions  with  the  best 
matching  parts  of  the  same  image,  carefully  choosing  the  or¬ 
der  in  which  the  missing  region  is  filled  to  minimize  artifacts 
[8].  Instead,  we  propose  an  inpainting  method  that  represents 
missing  regions  as  sparse  linear  combinations  of  other  regions 
in  the  same  image  (in  contrast  to  [9],  in  which  sparse  repre¬ 
sentations  on  standard  dictionaries,  such  as  wavelets,  are  em¬ 
ployed),  computed  by  minimizing  a  simple  functional. 

2.  SPARSE  LINEAR  COMBINATIONS  OF 
EXEMPLARS 

The  image  model  of  the  proposed  approach  is  that  each  block 
of  the  restored  image  should  be  a  sparse  linear  combination 
of  other  image  blocks,  either  from  known  regions  (i.e.  not  in¬ 
tersecting  the  inpainting  region)  of  the  image  being  restored, 
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Fig.  1.  Image  blocks  which  cross  the  boundary  of  the  inpaint¬ 
ing  region  can  be  chosen  as  the  best  match  to  the  image  model 
subject  to  the  constraint  that  they  match  the  known  image  pix¬ 
els  outside  the  inpainting  region,  but  in  a  tiled  block  structure, 
blocks  interior  to  the  inpainting  region  are  unconstrained.  In 
an  overlapping  block  structure,  an  additional  constraint  on  the 
mismatch  between  overlapping  blocks  allows  the  constraint 
from  the  known  pixels  to  propagate  to  interior  blocks. 

Within  this  framework,  the  solution  is  computed  by  mini¬ 
mizing  a  global  functional  which 

1 .  penalizes  the  i1  norm  of  the  linear  combination  coeffi¬ 
cients  to  encourage  a  sparse,  low  complexity,  solution, 

2.  constrains  (or  penalizes)  the  mismatch  between  solu¬ 
tion  blocks  and  known  pixels,  and 

3.  constrains  (or  penalizes)  the  mismatch  between  over¬ 
lapping  parts  of  different  solution  blocks. 

In  order  to  discuss  this  approach  in  more  detail,  we  need  to 
establish  some  notation.  Denote  the  image  to  be  inpainted 
by  vector  s,  the  inpainting  region  mask  by  r,  and  the  in¬ 
painted  result  by  u.  To  reduce  the  complexity  of  computing 
the  mismatch  between  overlapping  parts  of  different  blocks, 
the  blocks  are  arranged  in  indexed  grids,  with  the  overlap  be¬ 
ing  produced  by  an  offset  of  the  entire  grid,  as  indicated  in 
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block  is  indexed  by  the  number  of  its  grid  and  its  number 
within  that  grid,  block  s kj  being  the  Ith  block  in  grid  k. 


Fig.  2.  Structure  of  overlapping  block  grids. 

Denote  the  operator  that  extracts  block  k ,  l  from  an  image 
as  Bk:i  and  the  operator  that  “inserts”  the  same  block  back 
into  the  image  as  B^t,  and  make  the  definitions 

R  =  diag(r)  rk,i  =  Bkj  r 

Rk,i  =  diag(r k,i)  sk,i  =  RkjBkjs  . 

We  define  <frkj  and  atkj  as  the  dictionary  and  coefficients 
respectively  for  block  k,  l.  The  dictionary  is  indexed  because 
it  is  reasonable  (and  consistent  with  other  exemplar-based  im¬ 
age  restoration  algorithms)  to  choose  a  distinct  dictionary  for 
each  block  by  finding  similar  blocks  within  the  same  image, 
or  within  an  external  training  set  of  blocks  extracted  from 
similar  images.  For  the  initial  experiments  reported  here, 
however,  a  simpler  approach  is  adopted  in  which  all  blocks 
share  a  common  dictionary,  constructed  from  all  complete 
(i.e.  not  intersecting  the  inpainting  region)  blocks  within  a 
specified  training  region  in  the  image  to  be  inpainted.  This 
approach  remains  effective  since  finding  the  linear  combina¬ 
tion  with  minimum  i1  norm  has  the  effect  of  selecting  a  rep¬ 
resentation  in  terms  of  similar  blocks,  when  possible. 

Now,  for  each  block  we  wish  to  mini¬ 

mize  or  constrain  (either  equal  to  zero,  or  less  than  some  up¬ 
per  bound)  the  following  terms: 

1 .  S  olution  sparsity  1 1  a  k  5  / 1 1  % 

2.  Mismatch  with  known  pixels  in  s k,i 

\\Rk,i^k,iock,i  -  Rk,iBk:is\\2 

3.  Mismatch  in  overlap  with  block  grid  n 

||  ^k,l^-k,l  Rk,l  ^  ^  Rn,m^n,rn^n,rri  ||  2  Tl  ^  k 
m 


Indexed  symbols  are  combined  over  all  blocks  to  give  the  def¬ 
initions 
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Matrices  ^  and  B  have  the  same  diagonal  block  structure  (in 
terms  of  their  indexed  components)  as  R.  Note  that  these 
definitions  have  been  simplified  by  assuming  there  are  only 
two  block  grids,  but  in  principle  a  larger  number  is  possible, 
and  may  well  enhance  performance  at  the  expense  of  greater 
computational  cost. 

Our  global  penalties/constraints  may  then  be  written  as 

1 .  S  olution  sparsity  1 1  a  1 1  % 

2.  Mismatch  with  known  pixels  in  s 

||/M>a  —  s||2 


3.  Mismatch  between  overlapping  block  grids 

\\(I  -  BC)$<x\\2 


The  two  most  obvious  ways  of  combining  these  distinct  goals 
are  the  unconstrained  problem 

min  {||a||i  +70||iM>a:  -  s||2  +71IIU  “  BC)$a.\\2} 

with  weights  70  and  71 ,  and  the  constrained  problem 

min  || o: ||  1  s.t.  ||i^^>a  — s||2  <  cr0  and  || (I  —  BC)^ol\\2  <  cri 

with  upper  bounds  <to  and  g\.  For  the  preliminary  experi¬ 
ments  reported  here  we  concentrate  on  the  latter  problem,  but 
in  the  absence  of  an  efficient  solver  for  this  exact  problem,  we 
define 

/  70jR3>  \  .  _  /  7os  \ 

V  71  (I -BO*  J  {  0  ) 
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and  solve  the  related  problem 

min||a||i  such  that  \\Act  —  b||  <  a  , 

using  SPGL1  [10,  11].  The  parameters  70  and  71  balance 
the  influences  of  the  two  “mismatch”  components  of  the  con¬ 
straint,  and  a  determines  the  maximum  of  this  combined 
constraint.  The  final  restored  image  (or  image  region,  since 
the  computational  domain  can  be  restricted  to  a  small  re¬ 
gion  around  the  inpainting  region)  is  obtained  by  averaging 
the  overlapping  block  grids  to  obtain  a  single  value  for  each 
image  pixel. 

3.  RESULTS 

We  illustrate  the  performance  of  the  proposed  method  using 
two  small  test  regions  extracted  from  the  well-known  Lena 
and  Barbara  test  images,  and  compare  performance  with  the 
Field  of  Experts  method  [12]  and  the  method  of  Criminisi  et 
al.  [8],  computed  using  the  publicly  available  software  [13] 
and  [14]  respectively.  Results  for  the  first  region,  extracted 
from  the  hat  in  the  Lena  image,  are  displayed  in  Figure  3, 
and  the  results  for  the  second,  extracted  from  the  textured 
pants  in  the  Barbara  image,  are  displayed  in  Figure  4,  and 
a  comparison  for  SNR  values  is  displayed  in  Table  1.  For 
the  results  presented  here,  we  chose  parameters  empirically 
to  be  70  =  1.00,  71  =  0.01,  and  a  =  40.0.  In  each  case, 
a  subregion  of  the  image  was  extracted  for  training  (i.e.  as 
a  source  for  the  image  blocks  populating  the  dictionary),  and 
the  smaller  subregion  actually  displayed  was  tiled  by  the  over¬ 
lapping  block  grids  and  used  for  minimization  of  the  func¬ 
tional.  When  evaluating  the  performance  of  the  competing 
algorithms,  these  algorithms  were  presented  with  the  larger 
(training)  subregion  for  a  fair  comparison. 


Image 

FoE  [12] 

Criminisi  et  al.  [8] 

Proposed 

Lena 

15.4dB 

15.6dB 

17.8dB 

Barbara 

6.2dB 

2.3dB 

8.1dB 

Table  1.  Comparison  of  inpainting  SNR  values  on  test  images 
displayed  in  Figures  3  and  4. 

In  terms  of  subjective  image  quality,  the  FoE  reconstruc¬ 
tion  (which,  in  its  defense,  was  perhaps  designed  for  narrower 
inpainting  regions)  is  noticeably  inferior  to  the  other  two  on 
the  Lena  image,  and  very  much  inferior  on  the  Barbara  im¬ 
age.  (Despite  the  significant  subjective  quality  difference  on 
the  second  example,  the  FoE  SNR  is  significantly  higher  than 
that  of  the  method  of  Criminisi  et  al. ,  illustrating  the  limited 
utility  of  SNR  as  an  objective  quality  measure  in  this  context.) 
The  proposed  method  provides  slightly  better  subjective  im¬ 
age  quality  than  that  of  Criminisi  et  al.  for  the  Lena  image 
(more  clearly  visible  on  an  image  display  than  in  the  printed 
examples),  and  a  more  significant  performance  advantage  for 
the  Barbara  image.  For  these  comparisons,  runs  times  were 


similar  to  those  of  the  FoE  method,  and  2-3  orders  of  magni¬ 
tude  longer  than  the  method  of  Criminisi  et  al. . 

4.  CONCLUSIONS 

The  simple  initial  implementation  of  the  proposed  inpainting 
methods  exhibits  high  computational  cost,  but  delivers  com¬ 
parable  or  better  results  than  the  significantly  more  conceptu¬ 
ally  complex  methods  with  which  it  was  compared,  providing 
evidence  of  the  potential  of  the  proposed  approach.  A  number 
of  significant  improvements  remain  to  be  considered,  includ¬ 
ing  application  of  a  more  appropriate  solver  for  the  minimiza¬ 
tion  problem  (particularly  important  given  the  computational 
expense),  and  selection  of  optimized  dictionaries  for  each  so¬ 
lution  block. 
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